Method for sliced inpainting

ABSTRACT

A method for replacing image data in a destination region that is divided into sub-pieces along one or more cutting paths, which start and end at two different points on the border, and finding replacement data for the sub-pieces. The cutting paths may be determined as a function of the type of image structured at the start and the end points. The cutting paths may also be determined as a function of the area of the sub-pieces and the lengths of the cutting paths. Optionally, the destination region may be determined by a spot detection algorithm. Further optionally, the spot detection algorithm may comprise calculation of a high pass filter, or detection of areas of luminosity and border-to-volume ratios. A method for moving an image element within an image is also provided.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 15/260,390, filed Sep. 9, 2016 and titled METHOD FOR SLICINGINPAINTING, which is a continuation of U.S. patent application Ser. No.14/720,601, filed May 22, 2015 and titled METHOD FOR SLICED INPAINTING(now U.S. Pat. No. 9,443,285), which is a continuation of U.S. patentapplication Ser. No. 11/946,005, filed Nov. 27, 2007 and titled METHODFOR SLICED INPAINTING (now U.S. Pat. No. 9,053,530), which claims thebenefit of U.S. Provisional Patent Application No. 60/867,373, filedNov. 27, 2006 and titled Method for Sliced Inpainting, the content ofall of which are incorporated by reference in this disclosure in theirentirety.

BACKGROUND

Inpainting repairs defective, unwanted or missing image areas byreplacing them with a suitable, adapted piece found elsewhere in theimage, preferably without visible artifacts.

Many papers and patents have been published on inpainting. Inpaintingtypically covers the problem of replacing annoying, unwanted or missingimage data with new pixels. Some suggestions have been made to repairthe image area with repeated use of convolution kernels, while somesuggest or implement the use of replacement pixels or replacementstructure.

Routines work well only when the user provides a suggestion to thealgorithm as to which image area might make a good candidate forreplacing the area to be inpainted. Routines that try to findreplacement data automatically do not work well once the replacementarea is large (larger than 1% of the image). What is needed is a systemthat does a better job in finding replacement areas, and that canassemble replacement image data comprised of multiple smaller areas ifthe area to be inpainted is very large.

SUMMARY

The invention meets this need by providing a method for replacing imagedata in a destination region having a border, comprising dividing thedestination region into sub-pieces along one or more cutting paths, thecutting paths starting and ending at two different points on the borderof the destination region; and finding replacement data for thesub-pieces.

The cutting paths may be determined as a function of the type of imagestructured at the start and the end point of the cutting path. Inanother embodiment, the cutting paths are determined as a function ofthe area of the sub-pieces and the lengths of the cutting paths.

The destination region may be the product of a rotation, a perspectivecorrection, a lens distortion correction, or a creative imagedistortion. Optionally, the destination region may be determined by aspot detection algorithm. Optionally, the spot detection algorithm maycomprise calculation of a high pass filter, or detection of areas ofluminosity and border-to-volume ratios.

Also provided is a method for moving an image element within an image,comprising receiving the boundary of a source region comprising theimage element; receiving a destination region within the image; copyingimage data from the source region to the destination region; determininga residual region, comprising area of the source region that does notintersect with the destination region; and applying an inpaintingroutine to the residual region.

Optionally, before the applying step, the residual region may be dividedinto sub-pieces. The copying step may comprise an adaption method basedupon differences between pixel values of the margin of the source regionand the margin of the destination region.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIG. 1 shows various aspects of the invention disclosed. FIG. 1A depictshow a user can provide a region of interest to the system/application toindicate an image imperfection that needs to be removed. FIG. 1B depictsin simplified form a selected part of the image that was moved by theuser. FIG. 1C show rotated or distorted images which no longer fill outthe full rectangular shape of the image. FIG. 1D shows how a margin canbe split up into four sections. FIG. 1E shows the sections of FIG. 1Dwith a slight offset per section.

FIG. 2A depicts a sub-optimal copying of a replacement structure into adestination area, and FIG. 2B depicts an enhanced copying using a slightdistortion.

FIG. 3 demonstrates how a large object may be removed using anembodiment of the slicing aspect of the invention. FIG. 3A shows a largeobject that needs to be removed.

FIG. 3B shows an object reduction using one whole, un-sliced replacementpiece of a different area in the image, and FIG. 3C shows how goodresults can be obtained if the replacement structure is assembled ofmultiple smaller regions.

FIGS. 4A-4D depict an image and inpainting of the upper left corner ofthat image according to one embodiment of the invention.

FIG. 5 shows a process for slicing regions to be inpainted according toan embodiment of the invention. FIG. 5A shows a potential region ofinterest C, FIG. 5B shows a potential “slicing” of the region C intosub-regions D₁ . . . D₅, and FIG. 5C shows a cutting vector.

DETAILED DESCRIPTION

We inpaint an image area by the following routine:

Algorithm 01

-   -   1—receive pixels of area to be inpainted (area D)    -   2—find an area that has suitable structure (area S) to replace        area D    -   3—copy pixels of area S into area D    -   4—adapt the new pixels in area D so that they fit seamlessly

While the routine for step 3 is trivial, 1, 2 and 4 are not. Thereforethe routines in step 1, 2 and 4 are described below in detail; Step 1:“Finding Area D”; Step 2: “Finding Area S”; and Step 4: “Adaption”.

Step 1: Finding Area D

Finding Area D (the area to be inpainted) is not discussed much in theliterature since it is generally assumed that the user circles orotherwise identifies the unwanted area. In this disclosure we suggestsome other methods to obtain an area D.

Note that when the plural “we” is used in this disclosure it is notmeant to imply that there are multiple inventors. It is solely used as agrammatical device for explication.

Finding Area D—Method 1: User Defined

This approach is trivial: allow the user to brush, paint, circle orotherwise determine an image area. One of ordinary skill with referenceto this disclosure will know how to allow a user to point out a regionof interest. FIG. 1A shows how a user can circle an unwanted area 101 ina digital image with a pointing device 102. The pixels within the areaare then considered pixels within area D.

Finding Area D—Method 2: Automatic Dust and Blemish Detection

This method is suitable to identify dust, skin blemishes and othersimilar details that the user typically wants to remove. It can be doneby the following spot-detection-algorithm:

Algorithm 02

-   -   1—create a highpass of the image I and store it in H    -   2—Create H′ so that H′=0 if H<n and H′=1 else.    -   3—Find an area which isn't yet inpainted of 20 to 600 connected        1's in H′ that has a low border-pixel-to-total-pixel-count. END        if no such area can be found.    -   4—apply Algorithm 01 to the found area, the found area being D.    -   5—go to step 3.

We suggest that the highpass used in step 1 utilize a radius of about0.05% to 1.5% of the image diameter and produces a result the mean ofwhich is 128.

The variable n in step 2 can be any number lower than 128, for instance125. The lower the number, the stronger and contrastier the blemishesneed to be to be detected. Note that the algorithm assumes that unwantedobjects are darker that the surrounding image, which is typically thecase.

In step 3, the range 20 to 600 can of course be adapted to the imagesize and to the imperfections to be removed. For instance, it may beadjusted to the type of dust that may occur on the sensor. For instance,some SLR cameras may have larger dust than other cameras based on thephysics of the camera. Sufficient literature has been written on how todetect whether an area of connected 1's is prolonged, compact orcircular, particularly based upon calculations onborder-pixel-to-total-pixel ratio respectively outline to volume ratio.The optimal theoretical border-pixel-to-total-pixel ratio of an image is4a/a², which will due to noise not be reached, so 10a/a² is a good valueto start with for a low border-pixel-to-total-pixel-count.

Finding Area D—Method 3: Undefined Areas Due to Moved Selections

This is a rather helpful technique for users dealing with imageretouching. FIG. 1B depicts a selected bitmap area 202 containing animage element that the user has moved to the right, leaving pixels inbitmap area 201 behind that are no longer defined. Some image editingapplications leave white pixels or transparent pixels (holes) behind,others leave the original pixels behind. In any case, unwanted artifactscan occur. For instance, if the selected bitmap area 202 enclosed aball, the bitmap area 201 may now show half a ball after the moving ofthe bitmap portion (in those instances in which original pixels are letbehind) or a “hole” (in those instances in which original pixels arereplaced with some default value).

However, if the bitmap area 201 is defined as area D and algorithm 01 isapplied, said bitmap area 202 can be moved around in the image, leavingno artifacts or holes in bitmap area 201. Using this method, an imageelement can be moved within an image by receiving the boundary of asource region comprising the image element, receiving a destinationregion within the image, copying image data from the source region tothe destination region, determining a residual region, comprising areaof the source region that does not intersect with the destinationregion, and applying an inpainting routine to the residual region.

Finding Area D—Method 4: Undefined Areas Due to Image Distortion

When images are distorted (rotated, corrected for perspective, orcorrected for lens defects such as barrel or pincushion distortion) theytypically no longer have a shape that matches a straight rectangle thatis parallel to the x and y axes. Therefore significant parts of thedistorted image need to be cut away to obtain a rectangular image again,or undefined (white) pixels need to be included in the result. FIG. 1Cshow such a result. Rotated image 300 shows the result of a rotation.Distorted image 400 shows the result of a more complex distortion, forinstance a lens correction with a connected perspective correction. Thesmall enclosed areas along the border show the areas that would nolonger be defined.

Now, if all pixels along the margin of the resulting image 300 or image400 are filled up with the closest pixel that carries actual values,said undefined areas are surrounded by a margin of now defined pixels.If successively those areas (typically 4 areas) are defined as area Dand fed into Algorithm 01, and the four results are combined to oneimage, a distorted image of acceptable size without missing or undefinedpixels can be obtained. How to combine the four results into one imageis evident to those skilled in the art; during the merging definedpixels need to overwrite the undefined pixels.

FIG. 4 show such a progress. FIG. 4A shows a rotated image with four“empty” corners. FIG. 4B shows how margin pixels were filled in on theupper left corner to form a closed area C. FIG. 4C shows how thatdestination area C was split up (see below) into two closed areas D1 andD2, using a linear interpolation of pixel values along the cuttingpath/cutting vector. FIG. 4D shows the final inpainted result for theupper left corner.

Finding Area S

Algorithm 01 contains a step that requests finding an area S thatcontains a structure suitable to be inserted at the location of the areaD. (“S” stands for source and “D” for destination of the copy process.)Imagine that D contains some foliage on the ground, or patterned tileson a wall, or fine twigs against a sky. In that case that the finedetails along the margin of S and D should be as similar as possible.This Algorithm 03, which parses the details of the margin at a fineresolution, optimally on a pixel level, may be used to find S:

Algorithm 03

-   -   1—detect the pixels at δD, being the margin of D.    -   2—make a guess for a candidate for S and detect its margin        pixels, being S.    -   3—since δD and δS are vectors of many RGB vectors, measure the        quality of match by the Euclidean distance between the vectors        δD and S.    -   4—unless a high number (about 10,000) of candidates were tested,        go to 2    -   5—define S as the candidate with the best measured quality.

This algorithm leads to surprisingly good results, since a very detailedcomparison of the margin is performed. If higher speed is desired (thealgorithm isn't slow, though), a rougher scale can also be used, but werecommend not to downscale the image by more than a factor of 5.

Other distance measures could be tried. For example, the algorithm canbe enhanced by combining the measure for the quality of a candidate forS by measuring both the Euclidean vector distance and the angle betweenthe vectors, which makes the measure for the quality less dependent onabsolute brightness. Note that angle here refers to the angle betweentwo vectors in a space of several hundred dimensions, depending on theborder length of δD and S.

Also, the algorithm can be enhanced by applying Algorithm 03 on ahighpass version of the image, ensuring that absolute colors and softcolor gradients are disregarded in the measuring of the quality. Step 4of Algorithm 01 will correct for absolute color differences and lowfrequency mismatches. A further enhancement is to apply the algorithm ofa weighted mix of the original image and its highpass.

The algorithm can be even more enhanced by not applying it on RGB valuesbut on either Lab values (id est 3 channels), or even more channels suchas luminance, entropy, median of luminance, high pass of luminance, highpass of chroma a, or high pass of chroma b. This ensures even more thatreplacement areas are found that have not only small vector differencein details along the margin, but also a comparable structure, to avoid,for instance, foliage being replaced with tapestry, or grass beingreplaced with clouds.

Note that step 2 of Algorithm 03, “make a guess for a candidate for S”,refers to defining an area of the same size as D within the same imageor a different image. This can be done by using a random position, or bysuccessively moving it along a virtual and more or less coarse raster.If different images are taken into consideration, the algorithm mayfocus on comparable images on the hard drive with similar EXIF data, forinstance images with comparable ISO setting. Alternatively, considerthat the user has used a “remove skin blemish” tool or a “remove objectfrom wall” tool, etc., then S can also be searched in a database ofimages containing such structure.

Enhancement to Finding S

FIG. 1D shows the shape of a source margin 500, which is δS. Note thatthe shape of the source margin 500 and the destination margin δD (notshown) are at this time identical. The curve δS is partitioned intointervals 501, 502, 503, and 504. If δS is provided by aparameterization by its arc length s (which is a convenientrepresentation in the discrete pixel world), for instance so that thetrace of δS is given by δS: [0 . . . s]→R², then the four intervals canbe defined as

-   -   501: [s/4, s/4)    -   502: [s/4, s/2)    -   503: [s/2, 3s/4)    -   504: [3s/4, s]        having the center points s/8, 3s/8, 5s/8, and 7s/8 respectively.        Note that it is irrelevant where the starting point of the        parameterization is defined, but it might be slightly beneficial        to position one of the center points so that it coincides with a        stark contrast change on δD (again, keep in mind that δS and δD        have the same shape, so considerations on their        parameterizations are interchangeable).

Remember that Algorithm 03 describes that S can be found by overlayingsuccessively the margin δS over the image, measuring the quality ofmatch and defining S as the candidate that was found to have the bestmatch.

The algorithm can be enhanced by—whenever a good match is encounteredadditionally moving the four sections of the margin candidate intervals501, 502, 503, and 504 around by a small maximum spatial distance, sothat it is detected whether a slight offset may increase the match onone or more of the four sides. As shown in FIG. 1E, the temporarytorn-apart source margin 600 is now called δS*, with intervals 501, 502,503, and 504 now moved and represented by intervals 601, 602, 603, and604. What is actually overlaid over the image and tested for a qualityof match may look like what is shown in FIG. 1E.

If this leads to a better match, then the four center points atδS*(s/8), δS*(3s/8), δS*(5p/8), δS*(7p/8) (of the torn-apart sourcemargin 600) are stored as variables, the image area around the foundcandidate is also stored in a separate memory block which is thendistorted to match better. The easiest approach is to distort it so thatthe centers of the shifted curve segments of δS* match the centers ofthe four components of the undistorted curve δS. Note that a distortionbased upon four reference points is clearly defined, for instance knownas perspective distortion. We are distorting the entire found area(interior and its margin) to better match the destination area.

The in such way distorted bitmap and the trace of δS (not δS*) withinthis bitmap is then the found candidate for S. Out of such candidates,find the optimal one and define it as S. FIG. 2 shows the advantages ofsuch a method: If the margins of a replacement piece don't fit optimally(FIG. 2A, arrows), a slight distortion (FIG. 2B, arrows) can enhance theresult.

Bitmap Adaption

The last step of Algorithm 01 is to adjust the brightness of the copiedpixels. Imagine that this algorithm—when step 4 is reached—has copied amore or less suitable bitmap from S into D. If this is for instance apiece of tapestry in a shadow that was copied into a bright piece oftapestry, then by the end of step 3 there will be a dark area within theimage. This is a rather easy problem, since the pixels are more or lessuniformly too dark, but it may get more complicated: If foliage iscopied from here to there, the luminosity difference may be much morecomplicated due to the complex structure.

Bitmap Adaption—3×3 Kernel Adaption

This adaption method is based on the margin differences. Once the pixelshave been copied from S into D, the margin difference δM=δD−δS iscalculated (δD and δS referring to the original pixels at the margin ofthe two areas S and D in the original image. The subtraction refers to apairwise subtraction of pixel values, not an exclusion operator). Thedifference δM is then a number of border pixels having positive andnegative values. We suggest to create a (signed) pixel block M inmemory, and to write the values of 8M into this pixel block and then tofill the values in M based upon the border values in M.

As suggested by Oliveira et al, proceedings of VIIP 2001, “Fast DigitalInpainting”, M can be gained from δM by iteratively applying a kernelsuch as the following to the pixels enclosed by δM, and not modifyingthe border pixels themselves:

$\begin{matrix}{\quad\begin{matrix}\left\lbrack {0.073235,} \right. & {0.176765,} & 0.073235 \\{0.176754,} & {0.0,} & 0.176765 \\{0.073235,} & {0.176765,} & \left. 0.073235 \right\rbrack\end{matrix}} & \;\end{matrix}$

This method of adaption can also be used if a user manually copies animage element from one location to another, such as in FIG. 1B, wherethe element 202 may not fit seamlessly into its new location.

Bitmap Adaption—Illumination Adaption

Another approach is to determine the illumination function in the area Dand in the area S (before pixels were copied), named ill(D) and ill(S),and—after the copying has taken place—to adjust the illumination of thecopied pixels by multiplying them by the factor ill(D)/ill(S). Thoseskilled in the art will know that pixels can be represented by theproduct of a surface color and its illumination. Based on this theory,the above mentioned multiplication will bring the copied pixels into thecorrect light, making them match better.

In all suggested adaption methods, it may facilitate the implementationif the data is first copied from S to S, S′ being a region in a separatebitmap in memory, then δM and M are calculated, then the contents of Sis adapted using M, and then the contents of S′ is copied into the areaD.

Making source pixels fit seamlessly into a destination area by adding acorrection matrix M onto it can also be used if a user wants to copydata within an image or across an image. For instance, when bitmap dataare moved as shown in FIG. 1B, the bitmap data can be adapted to fit tothe new location by defining δM=δD−δS, S being the copy source area andD being the copy destination area, and filling in M using one of theabove named adaptions. The advantage of using, for instance, a kernel toiteratively fill M based upon δM is that the algorithm works even if δMis not continuous. This allows for bitmap adaption even if part of δM isnot defined, which may occur when a bitmap is copied on an image areathat is only partly defined—see the bitmap area 201 for such anundefined area.

Slicing D

The idea slicing D for better results is the idea to “divide andconquer”. Concretely, this can look as follows:

Algorithm 04

-   -   1—receive large area C    -   2—divide C into D₁, D₂ . . . D_(N), where D₁ . . . D_(N) are        disjoint and

D ₁

D ₁ . . .

uD _(N) =C

-   -   3—apply algorithm 01 on D₁, D₂ . . . D_(N)

Obviously, the difference is now that the replacement area is a little“patchwork” of multiple bitmaps. The advantage is that Algorithm 01 ismore likely to find well-fitting replacement areas when the receivedarea to be inpainted is not too large, and also that the inpainted areaC will not contain a large structure identical to a different area inthe image, which would be confusing to the viewer.

FIG. 5A shows such a large area C. Note that area C is extended (i.e.,it is long and narrow), which may be very often the case. This makes theslicing approach even more successful. The routine for slicing C into D₁. . . D_(N), as shown in FIG. 5B, can look as follows:

Algorithm 05

-   -   1—receive large area C.    -   2—find many (10,000) cutting vectors that start at random point        of δC and end at a different random point of δC.    -   3—out of these, sort all out those cross pixels not belonging to        C.    -   4—out of the remaining, sort all out that are too short (<1% of        arc length of δC).    -   5—within the remaining, find the ones where the conditions A1,        A2 and A3 (as defined below) are low.

Algorithm 05 is suitable to divide an area C into two sub-areas byidentifying a cutting vector. To those skilled in the art of computerscience and recursive programming it will be obvious that this can beused to further cut C into more sub-areas until sufficient sub-areas D₁. . . D_(N) are found, as in FIG. 5B. How many areas N are to be founddepends on the desired effect and the size of C. A good start for asub-area amount may be N=int(arc length of δC/250). The conditions A1,A2 and A3 of line 5 of algorithm 05 can be the following:

Condition A1:

If a and b, as shown in FIG. 5C, are the start and end points of acutting vector, that is to say a and b lie at the intersection points ofthe cutting vector and δC itself, then condition A1 is the ratio of thearc length of δC from point a to b (take the shortest of the two pathsfrom a to b along δC) to the vector length. For example, if A1 is high,the vector is short and cuts a decent part off of C. However, if A1 islow (close to 1), the area enclosed between the vector and δC cannot belarge, which makes the cutting vector not a good candidate.

Condition A2:

The second condition is any means to compare the similarity of pixelswithin δC around the point a and the point b. For instance, the tenpixels closest to a and the ten pixels closest to b can be compared withregard to the color difference, the chrominance difference, etc.Condition A2 helps finding cutting vectors the ends of which (a and b)lie within similar structure. This should avoid that one end of thevector lies on a foliage structure and the other end lies on a skystructure, leading to a suboptimal cutting vector. A2 can be a number,where 0 represents good similarity (sky-sky) and 10 represents poorsimilarity (sky-foliage). Multiple methods are known to the skilled inthe art for detection of a difference between two sets/vectors ofcolors.

Condition A3:

The third condition A3 can be a low number if both a and b are remotefrom an edge within δC, and high if a or b is close to an edge in C.This should ensure that the cutting vector does not cut C in thevicinity of an object edge, where artifacting would be most visible.

If all conditions A1, A2, A3 are low, the current candidate is a goodcandidate for a cutting vector.

Side-Note: In some cases it may occur that the area C is not continuousto begin with, for instance in cases as depicted in FIG. 1B, if bitmaparea 202 was moved only by a short distance and intersects bitmap area201. If C is not continuous, δC will consist of several closed paths. Itis suggested that each of the sub-components of C enclosed by a closedpath can be considered a “slice” D_(n) of C. In other words: If C isdivided to begin with, one should use this dividedness as a starting forthe “divide and conquer” approach.

Once area C is cut into N sub-areas D₁ . . . D_(N), it is easy to usealgorithm 04 to replace the data within C with new data. Note that alongthe cutting vectors, no pixel values may exist, therefore we suggestthat the pixel values along the cutting vectors be interpolated betweenthe start and the end point of the cut. This is trivial since this is aone-dimensional interpolation. This needs to be done to ensure that allD_(n) have borders with defined pixels.

We suggest that one actually copies a little more data for every D_(n)(for instance by expanding the areas D₁ . . . D_(N) by a few pixels) andthen blending them into one another with a soft transition (by using aGaussian kernel on the copy mask) or by using the technique of “minimumerror boundary” (as suggested by D. S. Wickramanayake and H. E. Bez andE. A. Edirisindhe, “Multi Resolution Texture Synthesis in WaveletTransform Domain.”).

Note that any technique of “minimum error boundary” will replace thestraight vectors by more or less curved paths. Also, in a simplerapproach, one may want to convert the straight path into a curved path,such as a sine curve, being less detectable to the eye of viewer.

The above described method has shown to produce superior results whenlarge areas in an image containing structure need to be inpainted.

All features disclosed in the specification, and all the steps in anymethod or process disclosed, may be combined in any combination, exceptcombinations where at least some of such features and/or steps aremutually exclusive. Each feature disclosed in the specification,including the claims, abstract, and drawings, can be replaced byalternative features serving the same, equivalent or similar purpose,unless expressly stated otherwise. Thus, unless expressly statedotherwise, each feature disclosed is one example only of a genericseries of equivalent or similar features.

This invention is not limited to particular hardware described herein,and any hardware presently existing or developed in the future thatpermits processing of digital images using the method disclosed can beused, including for example, a digital camera system.

A computer readable medium is provided having contents for causing acomputer-based information handling system to perform the stepsdescribed herein.

The term memory block refers to any possible computer-related imagestorage structure known to those skilled in the art, including but notlimited to RAM, processor cache, hard drive, or combinations of those,including dynamic memory structures. Preferably, the methods andapplication program interface disclosed will be embodied in a computerprogram (not shown) either by coding in a high level language.

Any currently existing or future developed computer readable mediumsuitable for storing data can be used to store the programs embodyingthe afore-described interface, methods and algorithms, including, butnot limited to hard drives, floppy disks, digital tape, flash cards,compact discs, and DVD's. The computer readable medium can comprise morethan one device, such as two linked hard drives. This invention is notlimited to the particular hardware used herein, and any hardwarepresently existing or developed in the future that permits imageprocessing can be used.

Any currently existing or future developed computer readable mediumsuitable for storing data can be used, including, but not limited tohard drives, floppy disks, digital tape, flash cards, compact discs, andDVD's. The computer readable medium can comprise more than one device,such as two linked hard drives, in communication with the processor.

1. (canceled)
 2. A computer-implemented method comprising: identifying aplurality of areas adjoining the borders of an image, wherein theplurality of areas are created by a misalignment of the borders of theimage with respect to a horizontal axis and a vertical axis; finding arespective set of replacement image data corresponding to each area ofthe plurality of areas, wherein each respective set of replacement imagedata is obtained from a respective source region of the image or of adifferent image; adapting pixels of each respective set of replacementimage data to visually fit the corresponding area of the plurality ofareas; and after the adapting, producing an updated image by replacingeach of the areas of the plurality of areas with the correspondingrespective set of replacement image data.
 3. The computer-implementedmethod of claim 2, further comprising modifying the image to cause themisalignment of the borders of the image with respect to the horizontalaxis and the vertical axis.
 4. The computer-implemented method of claim3, wherein modifying the image includes rotating the image to cause theborders of the image to become misaligned with the horizontal axis andthe vertical axis.
 5. The computer-implemented method of claim 3,wherein modifying the image includes modifying pixels of the image tocorrect perspective distortion in the image caused by an image capturedevice that captured the image.
 6. The computer-implemented method ofclaim 3, wherein modifying the image includes a modifying pixels of theimage to correct distortion in the image caused by lens defects of animage capture device that captured the image.
 7. Thecomputer-implemented method of claim 2, further comprising determiningmargin pixels at a margin of the image, wherein the margin pixels havepixel values based on one or more pixels closest to the margin pixels,wherein each of the plurality of areas includes one or more of themargin pixels that form a border of the respective area.
 8. Thecomputer-implemented method of claim 2, wherein producing the updatedimage includes overwriting undefined pixels in the plurality of areaswith the respective sets of replacement image data.
 9. Thecomputer-implemented method of claim 2, wherein adapting the pixelsincludes iteratively applying a kernel to particular pixels enclosed byborder pixels determined by differences between pixel values of marginsof the respective source region and the corresponding area.
 10. Thecomputer-implemented method of claim 1, wherein finding the respectiveset of replacement image data includes: dividing one or more of theareas of the plurality of areas into a plurality of sub-pieces along oneor more cutting paths, each cutting path starting and ending at tworespective points on a border of the one or more areas; and finding arespective subset of replacement image data corresponding to eachrespective sub-piece of the plurality of sub-pieces, wherein therespective subset of replacement image data is obtained from therespective source region of one of: the image or the different image.11. The computer-implemented method of claim 10, wherein adapting pixelsof each respective set of replacement image data includes adaptingpixels of the respective subsets of replacement image data to visuallyfit the corresponding sub-pieces of the plurality of sub-pieces,Preliminary Amendment and wherein producing the updated image includesreplacing the respective sub-pieces of the image with the correspondingsubsets of replacement image data.
 12. The computer-implemented methodof claim 10, wherein the adapting includes interpolating pixel valuesalong the one or more cutting paths between a start point and an endpoint of each of the one or more cutting paths to determine one or morepixel values along the one or more cutting paths in the one or moreareas.
 13. The computer-implemented method of claim 9, furthercomprising expanding one or more of the sub-pieces to include additionalpixels and blending colors of the additional pixels with pixels in theone or more sub-pieces.
 14. A system comprising: a storage device; andat least one processor operative to access the storage device andconfigured to perform operations comprising: detecting a plurality ofareas along borders of a rectangular image, wherein the plurality ofareas are created by a misalignment of the borders of the image withrespect to a horizontal axis and a vertical axis; finding a respectiveset of replacement image data corresponding to each area of theplurality of areas, wherein each respective set of replacement imagedata is obtained from a respective source region of the image or of adifferent image; adapting pixels of each respective set of replacementimage data to fit the corresponding area of the plurality of areas,wherein the adapting is based upon differences between pixel values of amargin of the respective source region and a margin of the correspondingarea; and after the adapting, producing an updated image by replacingeach of the areas of the plurality of areas with the correspondingrespective set of replacement image data.
 15. The system of claim 14wherein the at least one processor is configured to perform furtheroperations comprising modifying the image to cause the misalignment ofthe borders of the image with respect to the horizontal axis and thevertical axis.
 16. The system of claim 15 wherein the operation ofmodifying the image includes rotating the image to causes the borders ofthe image to become misaligned with the horizontal axis and the verticalaxis.
 17. The system of claim 15 wherein the operation of modifying theimage includes modifying pixels of the image to correct distortion inthe image caused by an image capture device that captured the image. 18.The system of claim 14 wherein the operation of producing the updatedimage includes overwriting undefined pixels in the plurality of areaswith the respective sets of replacement image data.
 19. The system ofclaim 14 wherein the operation of finding the respective set ofreplacement image data includes: dividing one or more of the areas ofthe plurality of areas into a plurality of sub-pieces along one or morecutting paths, the one or more cutting paths starting and ending at tworespective points on a border of the one or more areas; and finding arespective subset of replacement image data corresponding to eachrespective sub-piece of the plurality of sub-pieces, wherein therespective subset of replacement image data is obtained from therespective source region of one of: the image or the different image.20. A non-transitory computer readable medium having stored thereonprogram instructions that, when executed by a processor, cause theprocessor to perform operations including: modifying the image to causemisalignment of borders of an image with respect to a horizontal axisand a vertical axis; identifying a plurality of border areas adjoiningthe borders of the image, wherein the plurality of border areas havepixels with undefined pixel values with respect to pixel values of theimage, wherein the plurality of border areas are created by themisalignment of the borders of the image; finding a respective set ofreplacement image data corresponding to each border area of theplurality of border areas, wherein each respective set of replacementimage data is obtained from a respective source region of the image orof a different image; adapting pixels of each respective set ofreplacement image data to visually fit the corresponding border regionof the plurality of border regions; and after the adapting, producing anupdated image by replacing each of the border areas of the plurality ofborder areas with the corresponding respective set of replacement imagedata.
 21. The non-transitory computer readable medium of claim 20wherein modifying the image includes at least one of: rotating the imageto cause the borders of the image to become misaligned with thehorizontal axis and the vertical axis; or modifying pixels of the imageto correct distortion in the image caused by an image capture devicethat captured the image.